Robot device controller for controlling position of robot

ABSTRACT

A first characteristic portion of a first workpiece and a second characteristic portion of a second workpiece are previously determined. A characteristic amount detection unit detects a first characteristic amount related to the position of the first characteristic portion and a second characteristic amount related to the position of the second characteristic portion in an image captured by a camera. A calculation unit calculates, as a relative position amount, the difference between the first characteristic amount and the second characteristic amount. A command generation unit generates a movement command for operating a robot based on a relative position amount in the image captured by the camera and a relative position amount in a predetermined reference image.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a Continuation application of U.S. Pat. ApplicationNo. 16/923,485, filed Jul. 8, 2020, which claims benefit of priorityfrom Japanese Patent Application No. 2019-219104, filed Dec. 3, 2019,and Japanese Patent Application No. 2019-142418, filed Aug. 1, 2019. Thedisclosures of these applications are incorporated herein by referencein their entirety for all purposes.

BACKROUND OF THE INVENTION Field of the Invention

The present invention relates to a robot device controller whichcontrols the position of a robot.

Description of the Related Art

In a robot device equipped with a robot, an operation tool that issuitable for a desired operation is attached to the robot so that thedesired operation can be performed. For example, a hand, as an operationtool, which grasps a workpiece is attached to the robot so that therobot device can convey the workpiece to a desired position.

The robot device can attach a workpiece grasped by the robot device toanother workpiece and arrange the workpiece inside another workpiece bycontrolling the position and orientation of the workpiece when conveyingthe workpiece. When performing such an operation, it is preferable tostrictly align the workpiece grasped by the robot device with anotherworkpiece. For example, in a case in which an operation for fitting oneworkpiece to another workpiece is performed, the operation may fail ifthe position and orientation of one workpiece deviate from the positionand orientation of another workpiece.

In prior arts, it is known that, when one workpiece is attached toanother workpiece, the position of the robot is controlled using animage captured by a camera. For example, a goal image of one workpieceor another workpiece can be prepared. The camera captures an image of aworkpiece when the robot device conveys the workpiece. Control foradjusting the position of the robot by comparing the image of theworkpiece with the goal image is known (see, for example, JapaneseUnexamined Patent Publication No. 2013-180380A and Japanese UnexaminedPatent Publication No. 2015-214022A).

Further, control for previously calibrating a visual sensor coordinatesystem with respect to a robot coordinate system and calculating thethree-dimensional position of a workpiece based on the position of theworkpiece in the visual sensor coordinate system is known.Alternatively, a Jacobian matrix related to the position and size of acharacteristic portion in an image can be previously generated. Controlfor correcting the position of a robot based on the position of acharacteristic portion in an image captured by a camera, the position ofthe characteristic portion in the target data, and the Jacobian matrixis known (for example, see Japanese Unexamined Patent Publication No.2017-170599A).

SUMMARY OF THE INVENTION

In a method which uses a Jacobian matrix so as to adjust the position ofa robot, the movement amount of the robot can be calculated bymultiplying the difference between the characteristic amounts ofcharacteristic portions in an image by the Jacobian matrix. However, theJacobian matrix may not be calculated accurately due to, for example, ameasurement error in calculation of the Jacobian matrix. Further, theJacobian matrix is calculated based on the position gap in the visualsensor coordinate system when the position of the robot is moved by aminute amount. Thus, the robot can be accurately driven in the vicinityof each characteristic portion. However, the accuracy decreases as therobot moves away from the characteristic portion. For example, there itthe problem that the accuracy is low when the robot is moved by a largemovement amount. As a result, the position of the robot may not beaccurately controlled.

In a method for calibrating a visual sensor coordinate system, thethree-dimensional position of the characteristic portion can be detectedbased on an image captured by a two-dimensional camera. However, in thismethod, it is necessary to previously calibrate the position of thevisual sensor coordinate system with respect to the robot coordinatesystem.

One aspect of the present disclosure is a robot device controller whichcause a robot to adjust the relative position of a second member withrespect to a first member by moving the second member. The controllerincludes a visual sensor which captures images of the first member andthe second member. The controller includes an operation control unitwhich transmits a command to the robot so as to drive the robot, and animage processing unit which processes images captured by the visualsensor. A first characteristic portion for detecting the position of thefirst member and a second characteristic portion for detecting theposition of the second member are previously determined. The imageprocessing unit includes a characteristic amount detection unit whichdetects a first characteristic amount related to the position of thefirst characteristic portion and a second characteristic amount relatedto the position of the second characteristic portion in the imagecaptured by the visual sensor. The image processing unit includes acalculation unit which calculates, as a relative position amount, thedifference between the first characteristic amount and the secondcharacteristic amount. The image processing unit includes a commandgeneration unit which generates a movement command for operating therobot. A relative position amount in a reference image including imagesof the first characteristic portion and the second characteristicportion when the second member is arranged at a target position withrespect to the first member is determined. The command generation unitgenerates a movement command for operating the robot so that the secondmember is arranged at the target position with respect to the firstmember, based on a relative position amount in an image captured by thevisual sensor and a relative position amount in the reference imageincluding the images of the first characteristic portion and the secondcharacteristic portion. The operation control unit changes the positionof the robot based on the movement command.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic view of a first robot device in an embodiment.

FIG. 2 is an enlarged perspective view of a first workpiece, a secondworkpiece, and a hand in the first robot device.

FIG. 3 is a block diagram of the first robot device in the embodiment.

FIG. 4 is a first flowchart of first control of the first robot devicein the embodiment.

FIG. 5 is a reference image of the first robot device in the embodiment.

FIG. 6 is a second flowchart of the first control of the first robotdevice in the embodiment.

FIG. 7 shows an image in which the second workpiece is misaligned withthe first workpiece.

FIG. 8 shows another image in which the second workpiece is misalignedwith the first workpiece.

FIG. 9 is an enlarged perspective view of a first workpiece, a secondworkpiece, and a hand in a second robot device in the embodiment.

FIG. 10 is an enlarged perspective view of the first workpiece, thesecond workpiece, and the hand when the orientation of the secondworkpiece deviates from that of the first workpiece, in the second robotdevice.

FIG. 11 shows an image captured by a first camera when the orientationof the second workpiece deviates from that of the first workpiece.

FIG. 12 shows an image captured by a second camera when the orientationof the second workpiece matches with that of the first workpiece.

FIG. 13 is an image captured by the second camera when the orientationof the second workpiece deviates from that of the first workpiece.

FIG. 14 is a flowchart of third control in the second robot device.

FIG. 15 is an enlarged perspective view of a first workpiece, a secondworkpiece, and a hand in a third robot device in the embodiment.

FIG. 16 is an enlarged perspective view of a first workpiece, a secondworkpiece, and a hand in a fourth robot device in the embodiment.

FIG. 17 is a schematic view of a fifth robot device in the embodiment.

FIG. 18 is an enlarged schematic view of a case, a workpiece, and a handwhen the workpiece is aligned with the case.

FIG. 19 is an enlarged schematic view of the case, the workpiece, andthe hand when the workpiece is misaligned with the case.

FIG. 20 shows an image in which the workpiece is misaligned with thecase.

DETAILED DESCRIPTION OF THE INVENTION

A robot device controller in an embodiment will be described withreference to FIGS. 1 to 20 . In the present embodiment, a robot devicefor assembling a product and a robot device for arranging a workpieceinside a case will be taken as examples and described.

FIG. 1 is a schematic view of a first robot device in the presentembodiment. The first robot device 5 includes a hand 2 as an operationtool (end effector) and a robot 1 which moves the hand 2. The firstrobot device 5 attaches a second workpiece 91 as a second member to afirst workpiece 81 as a first member.

The robot 1 is an articulated robot including a plurality of joints. Therobot 1 includes a base part 14 and a swivel base 13 supported by thebase part 14. The base part 14 is secured to the installation surface.The swivel base 13 is formed to rotate with respect to the base part 14.The robot 1 includes an upper arm 11 and a lower arm 12. The lower arm12 is rotatably supported by the swivel base 13 via a joint. The upperarm 11 is rotatably supported by the lower arm 12 via a joint. Further,the upper arm 11 rotates about its rotation axis parallel to theextending direction of the upper arm 11.

The robot 1 includes a wrist 15 coupled to an end of the upper arm 11.The wrist 15 is rotatably supported by the upper arm 11 via a joint. Thewrist 15 includes a flange 16 which is formed to rotate. The hand 2 issecured to the flange 16. The robot 1 of the present embodiment has sixdrive axes, but the embodiment is not limited to this. Any robot whichcan move an operation tool can be adopted.

The hand 2 is an operation tool for grasping and releasing the workpiece91. The hand 2 has a plurality of claw parts 3. The hand 2 is formed toopen and close the claw parts 3. The claw parts 3 sandwich the workpiece91 so as to grasp the workpiece 91. The hand 2 of the first robot device5 has the claw parts 3, but the embodiment is not limited to this. Asthe hand, any configuration which is formed to be able to grasp aworkpiece can be adopted. For example, a hand which grasps a workpieceby suction or magnetic force may be adopted.

The robot device 5 in the present embodiment includes a conveyor 75 as acarrier which conveys the first workpiece 81 to the robot 1. The carrieris arranged around the robot 1. The conveyor 75 is formed to convey theworkpiece 81 to a predetermined position. The conveyor 75 is formed toconvey the workpiece 81 at a predetermined movement speed.

In the first robot device 5 of the present embodiment, the robot 1attaches the workpiece 91 to the workpiece 81 while the conveyor 75continues to convey the workpiece 81. In other words, the workpiece 81is moved by the conveyor 75 during an operation for attaching theworkpiece 91. The robot 1 attaches the workpiece 91 to the workpiece 81while changing its position and the orientation so as to follow theworkpiece 81.

FIG. 2 is an enlarged perspective view of the first workpiece, thesecond workpiece, and the hand in the first robot device. Referring toFIGS. 1 and 2 , the second workpiece 91 has a grip part 94 whichprojects from the surface of a main body. The claw parts 3 grasp thegrip part 94 so that the hand 2 grasps the second workpiece 91.

The first workpiece 81 has projection parts 82 and 83 which project fromthe surface of the main body. The projection part 82 and the projectionpart 83 are spaced from each other. Further, the workpiece 81 issupported by the conveyor 75 so that the projection part 82 and theprojection part 83 are arranged in the vertical direction. Holes 82 aand 83 a are formed in the upper surfaces of the projection parts 82 and83, respectively. The second workpiece 91 has projection parts 92 and 93which project from the surface of the main body. Pins 92 a and 93 a aresecured to the projection parts 92 and 93, respectively. The pin 92 aand the pin 93 a are arranged linearly. In first control performed bythe first robot device 5, the pin 92 a is inserted into the hole 82 aand the pin 93 a is inserted into the hole 83 a.

In the first control of the present embodiment, the pins 92 a and 93 aare controlled so as to be aligned with the holes 82 a and 83 a. Theorientation of the workpiece 91 with respect to the workpiece 81 isadjusted before the first control is performed. In other words, theorientation of the robot 1 is adjusted so that the pins 92 a and 93 a ofthe workpiece 91 are arranged on a straight line extending in thevertical direction. Thus, in the first control, the control forarranging the pin 92 a directly above the hole 82 a is performed so thatthe pins 92 a and 93 a can be aligned with the holes 82 a and 83 a.After the first control of the present embodiment, a downward movementof the workpiece 91 in the vertical direction indicated by arrow 103causes the pins 92 a, 93 a to be inserted into the holes 82 a, 83 a, andcauses the workpiece 91 to be attached to the workpiece 81.

The robot device 5 includes a camera 25 as a visual sensor whichcaptures images of the first workpiece 81 and the second workpiece 91.The camera 25 in the present embodiment is a two-dimensional camera. Thecamera 25 is supported by the hand 2 via the support member 17. Thecamera 25 in the robot device 5 changes its position and orientationtogether with the hand 2.

The camera 25 captures an image when the second workpiece 91 approachesthe first workpiece 81. The camera 25 is arranged so as to capture animage of the vicinity of the portion in which the second workpiece 91 isengaged with the first workpiece 81. Further, the camera 25 is arrangedso as to be able to capture an image of a first characteristic portionfor detecting the position of the first workpiece 81 and an image of asecond characteristic portion for detecting the position of the secondworkpiece 91. In the robot device 5, the camera 25 is arranged so as tocapture images of the workpieces 81 and 91 from the position above theworkpiece 81 and the workpiece 91.

A reference coordinate system 51 is set in the robot device 5. In theexample shown in FIG. 1 , the origin of the reference coordinate system51 is arranged in the base part 14 of the robot 1. The referencecoordinate system 51 is also referred to as a world coordinate system.The reference coordinate system 51 is a coordinate system in which theposition of the origin is fixed, and the directions of the coordinateaxes are fixed. Even when the position and orientation of the robot 1change, the position and direction of the reference coordinate system 51do not change. The reference coordinate system 51 has, as coordinateaxes, an X-axis, a Y-axis, and a Z-axis which are orthogonal to oneanother. Further, a W-axis is set as a coordinate axis around theX-axis. A P-axis is set as a coordinate axis around the Y-axis. AnR-axis is set as a coordinate axis around the Z-axis.

The position and orientation of the robot 1 can be represented by thereference coordinate system 51. For example, the position of the robot 1can be represented by the position of a tool tip arranged at the tip ofthe hand 2. Further, a tool coordinate system which moves together withthe hand 2 can be set at the tool tip. The orientation of the robot 1can be represented by the direction of the tool coordinate system withrespect to the reference coordinate system 51.

FIG. 3 is a block diagram of the robot device according to the presentembodiment. Referring to FIGS. 1 to 3 , the robot 1 includes a robotdrive device which changes the position and orientation of the robot 1.The robot drive device includes a robot drive motor 22 which drivescomponents such as an arm and a wrist. The robot drive motor 22 isdriven and thereby causes the direction of each component to change.

The hand 2 includes a hand drive device which drives the hand 2. Thehand drive device includes a hand drive motor 21 which drives the clawparts 3 of the hand 2. The hand drive motor 21 is driven and therebycauses the claw parts 3 of the hand 2 to open or close. Note that theclaw parts may be formed to be actuated by air pressure. In thisrespect, the hand drive device may include a device such as an air pumpand a cylinder which drives the claw parts by air pressure.

A controller 29 for the robot device 5 includes a robot controller 4which controls the robot 1 and the hand 2. The robot controller 4includes an arithmetic processing device (computer) having a CPU(Central Processing Unit) as a processor. The arithmetic processingdevice includes, for example, a RAM (Random Access Memory) and a ROM(Read Only Memory) connected to the CPU via a bus. An operation program41, which is previously created so as to control the robot 1, the hand2, and the conveyor 75, is input to the robot controller 4. The robot 1and the hand 2 convey the workpiece 91 based on the operation program41. The conveyor 75 conveys the workpiece 81 based on the operationprogram 41.

The arithmetic processing device of the robot controller 4 includes astorage part 42 which stores predetermined information. The storage part42 stores information related to control of the robot 1, the hand 2, andthe conveyor 75. The storage part 42 can be comprised of a storagemedium capable of storing information, such as a volatile memory, anon-volatile memory, or a hard disk. The operation program 41 is storedin the storage part 42. The robot controller 4 includes a display 46which displays any information related to the robot device 5. Thedisplay 46 includes, for example, a liquid crystal display panel.

The arithmetic processing device includes an operation control unit 43which transmits an operation command for the robot 1 and the hand 2. Theoperation control unit 43 corresponds to the processor driven accordingto the operation program 41. The operation control unit 43 is formed soas to be able to read the information stored in the storage part 42. Theprocessor functions as the operation control unit 43 by reading theoperation program 41 and performing a control operation determined inthe operation program 41.

The operation control unit 43 transmits an operation command for drivingthe robot 1 to the robot drive part 45 based on the operation program41. The robot drive part 45 includes an electric circuit which drivesthe robot driving motor 22. The robot drive part 45 supplies electricityto the robot drive motor 22 based on the operation command. Theoperation control unit 43 also transmits an operation command fordriving the hand 2 to the hand drive part 44 based on the operationprogram 41. The hand drive part 44 includes an electric circuit whichdrives the hand drive motor 21. The hand drive part 44 supplieselectricity to the hand drive motor 21 based on the operation command.Furthermore, the camera 25 is connected to the robot controller 4 of thepresent embodiment. The operation control unit 43 transmits a commandfor capturing an image to the camera 25 based on the operation program41.

The arithmetic processing device of the robot controller 4 includes animage processing unit 31 which processes an image captured by the camera25. The image processing unit 31 has a characteristic amount detectionunit 32 which detects a characteristic amount of a characteristicportion which is a predetermined distinctive portion in each of theworkpieces 81 and 91. The image processing unit 31 includes acalculation unit 33 which calculates, as a relative amount, thedifference between the characteristic amount of the first workpiece 81and the characteristic amount of the second workpiece 91. The imageprocessing unit 31 has a command generation unit 34 which generates amovement command for operating the robot 1 based on the relative amountcalculated by the calculation unit 33.

The image processing unit 31 corresponds to a processor which is drivenaccording to the operation program 41. In particular, each unit of thecharacteristic amount detection unit 32, the calculation unit 33, andthe command generation unit 34 corresponds to a processor which isdriven according to the operation program 41. The processor functions aseach unit by reading the operation program 41 and executing a controloperation determined in the operation program 41.

The robot device 5 includes a state detector which detects the operationstate of the robot device 5. The state detector of the presentembodiment includes a position detector 23 which detects the positionand orientation of the robot 1. The position detector 23 is attached tothe robot drive motor 22 for the drive axis of a component such as anarm. For example, the position detector 23 detects a rotation angle atwhich the robot drive motor 22 is driven. The position and orientationof the robot 1 are detected based on the output of the position detector23.

The controller 29 for the robot device 5 includes a conveyor controller76 which controls the operation of the conveyor 75. The conveyorcontroller 76 includes an arithmetic processing device (computer)including, for example, a CPU and a RAM. The conveyor controller 76 isformed so as to be able to mutually communicate with the robotcontroller 4. The operation controller 43 transmits an operation commandfor driving the conveyor 75 to the conveyor controller 76 based on theoperation program 41. The conveyor controller 76 receives the operationcommand from the robot controller 4 and drives the conveyor 75.

The controller 29 for the robot device 5 according to the presentembodiment includes the robot controller 4 which controls the robot 1and the hand 2, and the conveyor controller 76 which controls theconveyor 75, but the embodiment is not limited to this. The robot device5 may be formed so that one controller controls the robot 1, the hand 2,and the conveyor 75.

Further, in the controller 29 for the robot device 5 according to thepresent embodiment, the robot controller 4 includes the image processingunit 31 having a function for processing an image, but the embodiment isnot limited to this. The controller for the robot device may include animage processing device (computer) having the image processing unit 31.The image processing device can be composed of an arithmetic processingdevice having a CPU as a processor. The processor for the imageprocessing device functions as a characteristic amount detection unit, acalculation unit, and a command generation unit, which are driven basedon the operation program. The image processing device is formed so as tobe able to mutually communicate with the robot controller.

FIG. 4 is a first flowchart of the first control in the presentembodiment. Referring to FIGS. 1, 2, and 4 , the first control causesthe workpiece 91 to be aligned with the workpiece 81 conveyed by theconveyor 75, by using an image captured by a single camera 25. Asdescribed above, the orientation of the workpiece 91 with respect to theorientation of the workpiece 81 is previously adjusted before the firstcontrol. In other words, the orientation of the robot 1 is adjustedbefore the camera 25 captures an image. In this control, the pin 92 a isaligned with the hole 82 a.

In the present embodiment, a relative position amount in the referenceimage is calculated before the robot device 5 actually performs anoperation. The robot device 5 then performs the operation using thepreviously calculated relative position amount of the reference image.The first flowchart shows a control operation for calculating therelative position amount in the reference image.

At step 111, a reference image for performing the first control isgenerated. FIG. 5 shows a reference image for performing the firstcontrol of the present embodiment. The reference image 61 corresponds toan image captured by the camera 25 when the second workpiece 91 isarranged at the target position with respect to the first workpiece 81.In the present embodiment, the reference image 61 is an image capturedby the camera 25 when the pins 92 a and 93 a of the workpiece 91 arearranged directly above the holes 82 a and 83 a of the workpiece 81. Thereference image 61 can be prepared by an operator and stored in thestorage part 42.

In the present embodiment, a first characteristic portion for detectingthe position of the first workpiece 81 is previously determined. In thefirst control, the upper surface of the projection part 82 is set as thefirst characteristic portion. Further, a second characteristic portionfor detecting the position of the second workpiece 91 is previouslydetermined. In the first control, the upper surface of the projectionpart 92 is set as the second characteristic portion. The characteristicportion is the portion in which a shape can be detected when an image isanalyzed. As a characteristic portion, a part of a workpiece, a patternformed on the surface of the workpiece, a line or a drawing depicted onthe surface of the workpiece, or the like can be adopted. Further, it ispreferable that a characteristic portion is set in the vicinity of acontact portion between the second workpiece 91 and the first workpiece81.

In the present embodiment, a set point P1 is set on the upper surface ofthe projection part 82 serving as the first characteristic portion inthe first workpiece 81. The set point P1 is set at a corner of theprojection part 82. The position of the set point P1 corresponds to theposition of the workpiece 81. The set point P2 is arranged on the uppersurface of the projection part 92 as the second characteristic portionin the second workpiece 91. The set point P2 is set at a corner of theprojection part 92. The set point P2 corresponds to the position of theworkpiece 91. The set points P1 and P2 are set in regions included inthe image captured by the camera 25.

Referring to FIGS. 3, 4, and 5 , at step 112, the characteristic amountdetection unit 32 of the image processing unit 31 detects the firstcharacteristic portion and the second characteristic portion of thereference image 61. In a method for detecting a characteristic portion,a base image which serves as a criterion for each of the workpieces 81and 91 can be prepared. A characteristic portion in the image capturedby the camera 25 can be detected by a method such as template matching,using the base image and the image captured by the camera 25. In thisexample, the upper surface of the projection part 82 and the uppersurface of the projection part 92 can be detected.

At step 113, the characteristic amount detection unit 32 detects thefirst characteristic amount related to the position of the firstcharacteristic portion and the second characteristic amount related tothe position of the second characteristic portion. In the presentembodiment, a screen coordinate system 52 is set for the image capturedby the camera 25. The screen coordinate system 52 is a coordinatesystem, the origin of which is set at any point in the image. The screencoordinate system 52 has a u-axis and a v-axis which are orthogonal toeach other. The screen coordinate system 52 corresponds to a visualsensor coordinate system of the camera 25.

The characteristic amounts related to the positions in the presentembodiment are a coordinate value of the u-axis and a coordinate valueof the v-axis of the screen coordinate system 52 in the image. Thecharacteristic amount detection unit 32 can detect, based oncharacteristic portions detected in the reference image 61, thepositions of the set points P1 and P2 which are set in thecharacteristic portions. The characteristic amount detection unit 32detects, as the first characteristic amount, a coordinate value (u1b,v1b) of the set point P1 on the screen coordinate system 52. Thecharacteristic amount detection unit 32 also detects, as the secondcharacteristic amount, a coordinate value (u2b, v2b) of the set point P2on the screen coordinate system 52.

Subsequently, at step 114, the calculation unit 33 of the imageprocessing unit 31 calculates a relative amount related to the firstcharacteristic amount and the second characteristic amount in thereference image. The calculation unit 33 calculates a relative positionamount as a relative amount in order to control the position of therobot 1. The relative position amount is a difference between the firstcharacteristic amount and the second characteristic amount. For example,the calculation unit 33 calculates, as the relative position amount, adifference (u1b-u2b, v1b-v2b) between the coordinate value of the firstcharacteristic amount and the coordinate value of the secondcharacteristic amount. The relative position amount in the referenceimage 61 calculated by the calculation unit 33 is stored in the storagepart 42 as a reference relative position amount.

In this way, the image processing unit 31 can calculate the relativeposition amount in the reference image 61. Note that, in the presentembodiment, the relative position amount in the reference image 61 ispreviously calculated and stored in the storage part 42. However, theembodiment is not limited to this. The relative position amount in thereference image 61 may be calculated each time the first control isperformed.

The reference image 61 does not have to be an image actually captured bythe camera 25. For example, three-dimensional data of each of theworkpieces 81 and 91 can be generated by, for example, a CAD (ComputerAided Design) device. It is possible to generate three-dimensional datawhen the workpiece 91 is arranged at the target position with respect tothe workpiece 81. The reference image 61 may be generated by projectingthe three-dimensional data on one plane in the direction correspondingto the orientation of the camera.

Subsequently, the robot controller 4 performs a control operation formoving the pin 92 a of the second workpiece 91 closer to the hole 82 aof the first workpiece 81 so that the first characteristic portion andthe second characteristic portion are arranged inside an imaging range25 a of the camera 25. This control operation can be performed by anycontrol operation. For example, the robot controller 4 detects theposition of the workpiece 81 on the conveyor 75 by using a predeterminedsensor. The robot controller 4 detects the position of the workpiece 81based on the movement speed of the conveyor 75. The robot controller 4can control the position and orientation of the robot 1 so that theworkpiece 91 moves closer to the workpiece 81.

FIG. 6 is a second flowchart of the first control in the presentembodiment. Referring to FIGS. 3 and 6 , after the robot controller 4performs a control operation for moving the workpiece 91 closer to theworkpiece 81, at step 115, the operation control unit 43 causes thecamera 25 to capture images of the workpieces 81 and 91.

FIG. 7 shows an image captured by the camera in order to adjust theposition of the second workpiece with respect to the first workpiece.The image 62 includes an image of the upper surface of the projectionpart 82 which is the first characteristic portion and an image of theupper surface of the projection part 92 which is the secondcharacteristic portion. In the image 62, the second workpiece 91 isdisplaced from the first workpiece 81 toward the positive side of theu-axis of the screen coordinate system 52 as designated by arrow 101.

FIG. 8 shows another image captured by the camera in order to adjust theposition of the second workpiece with respect to the first workpiece. Inthe image 63, the second workpiece 91 is displaced from the firstworkpiece 81 toward the negative side of the u-axis of the screencoordinate system 52 as designated by arrow 102. Referring to FIGS. 7and 8 , in the first control, a control operation for correcting such adisplacement of the second workpiece 91 is performed. Here, FIG. 7 ofFIGS. 7 and 8 will be taken as an example for description.

Referring to FIGS. 3, 6, and 7 , at steps 116 to 118, the same controlas the control for the reference image 61 is performed. The imageprocessing unit 31 detects the first characteristic amount and thesecond characteristic amount in the image 62 captured by the camera 25and calculates the relative position amount using the firstcharacteristic amount and the second characteristic amount.

At step 116, the characteristic amount detection unit 32 detects thefirst characteristic portion and the second characteristic portion ofthe image 62 captured by the camera 25. The upper surface of theprojection part 82 of the workpiece 81 is detected as the firstcharacteristic portion, and the upper surface of the projection part 92of the workpiece 91 is detected as the second characteristic portion.

At step 117, the characteristic amount detection unit 32 detects thefirst characteristic amount and the second characteristic amount in theimage 62 captured by the camera 25. The characteristic amount detectionunit 32 detects, as the first characteristic amount related to theposition of the first characteristic portion, a coordinate value (u1m,v1m) of the set point P1 on the screen coordinate system 52. Further,the characteristic amount detection unit 32 calculates, as the secondcharacteristic amount related to the position of the secondcharacteristic portion, a coordinate value (u2m, v2m) of the set pointP2 on the screen coordinate system 52.

At step 118, the calculation unit 33 calculates, as the relativeposition amount, the difference between the first characteristic amountand the second characteristic amount. The relative position amount inthe image 62 captured by the camera 25 is a difference (u1m-u2m,v1m-v2m) between the coordinate value of the first characteristic amountand the coordinate value of the second characteristic amount.

Note that, when the second workpiece 91 is grasped at a predeterminedposition of the hand 2, the second characteristic amount related to thesecond workpiece 91 is constant. Thus, the second characteristic amountmay be previously measured and stored in the storage part 42. In otherwords, the coordinate value of the set point P2 may be previously storedin the storage part 42. Meanwhile, the hand 2 may deviate from a desiredposition when grasping the second workpiece 91. For this reason, in thepresent embodiment, the second characteristic amount is also detected bya method for matching with the base image by template matching, based onthe actually captured image 62.

Subsequently, the command generation unit 34 of the image processingunit 31 generates a movement command for the robot 1 so that the secondworkpiece 91 is arranged at the target position with respect to thefirst workpiece 81, based on the relative position amount in the image62 captured by the camera 25 and the relative position amount in thereference image 61. The command generation unit 34 of the presentembodiment generates a movement command for operating the robot 1 sothat the relative position amount in the image 62 captured by the camera25 approaches the relative position amount in the reference image 61.

At step 119, the command generation unit 34 calculates the differencebetween the relative position amounts, which is the difference betweenthe relative position amount in the image 62 captured by the camera 25and the relative position amount in the reference image 61. In thepresent embodiment, the command generation unit 34 calculates thedifference between the relative position amounts by subtracting therelative position amount in the reference image 61 from the relativeposition amount in the image 62 captured by the camera 25. Thedifference between the relative position amounts can be represented by[(u1m-u2m)-(u1b-u2b), (v1m-v2m)-(v1b-v2b)] as a value for each of theu-axis and the v-axis. As described above, in the present embodiment,the difference between the relative position amounts related to theu-axis and the difference between the relative position amounts relatedto the v-axis are calculated.

Subsequently, at step 120, the command generation unit 34 determineswhether the difference between the relative position amounts remainswithin a predetermined determination range. The determination range ispreviously determined and stored in the storage part 42. For example,the determination range of the value related to the u-axis and thedetermination range of the value related to the v-axis can be preset.The closer the second workpiece 91 is to the target position withrespect to the first workpiece 81, the closer the difference between therelative position amounts becomes to zero. When the value related to theu-axis remains within the determination range and the value related tothe v-axis remains within the determination range, the differencebetween the relative position amounts can be determined to remain withinthe determination range. In other words, the command generation unit 34can determine that the alignment of the workpiece 91 with the workpiece81 has been completed.

Meanwhile, when at least either the value related to the u-axis or thevalue related to the v-axis deviates from the determination range, thecommand generation unit 34 can determine that the difference between therelative position amounts deviates from the determination range. Inother words, the command generation unit 34 can determine that theworkpiece 91 has not reached a desired position with respect to theworkpiece 81.

At step 120, when the difference between the relative position amountsremains within the determination range, the control ends. At step 120,when the difference between the relative position amounts deviates fromthe determination range, the control proceeds to step 121.

At step 121, the command generation unit 34 sets a driving method forthe robot 1 based on the difference between the relative positionamounts. The command generation unit 34 sets a movement direction and amovement amount of the position of the robot 1 in the referencecoordinate system 51. In the present embodiment, the movement directionof the position of the robot for the difference between the relativeposition amounts is previously determined. The movement direction of theposition of the robot 1 in reference coordinate system 51 is determinedwith respect to a positive value or a negative value of the u-axis ofthe screen coordinate system 52. For example, when the differencebetween the relative position amounts related to the u-axis is apositive value, the movement direction of (1, 1, 0) is previouslydetermined using the coordinate values of the X-axis, Y-axis, and Z-axisof the reference coordinate system 51. Further, when the differencebetween the relative positions related to the v-axis is a positivevalue, the movement direction of (0, 0, 1) is previously determinedusing the coordinate values of the X-axis, Y-axis, and Z-axis of thereference coordinate system 51.

Furthermore, a method for calculating the movement amount of theposition of the robot 1 with respect to the difference between therelative position amounts is previously determined. For example, as themovement amount of the position of the robot 1 in a directioncorresponding to the u-axis, a value obtained by multiplying a valuerelated to the u-axis ((u1m-u2m)-(u1b-u2b)) by a predeterminedcoefficient can be adopted. Further, as the movement amount of theposition of the robot 1 in a direction corresponding to the v-axis, avalue obtained by multiplying a value related to the v-axis((v1m-v2m)-(v1b-v2b)) by a predetermined coefficient can be adopted. Inthis way, the movement amount of the position of the robot 1 can becalculated in the direction corresponding to each axis of the screencoordinate system 52.

In the present embodiment, the movement amount in the X-axis direction,the movement amount in the Y-axis direction, and the movement amount inthe Z-axis direction in the reference coordinate system 51 arecalculated based on the difference between the relative position amountsrelated to the u-axis. Further, the movement amount in the X-axisdirection, the movement amount in the Y-axis direction, and the movementamount in the Z-axis direction in the reference coordinate system 51 arecalculated based on the difference between the relative position amountsrelated to the v-axis. Thus, in the reference coordinate system 51, twomovement amounts (a movement amount related to the u-axis and a movementamount related to the v-axis) may be calculated for one axis. In thisrespect, the position of the robot 1 may not move in the direction ofthe axis in which the two movement amounts are calculated.Alternatively, the final movement amount may be calculated bymultiplying each movement amount by a coefficient. Alternatively, eitherone of the movement amounts may be adopted.

Subsequently, at step 122, the robot 1 is driven based on the movementdirection and movement amount of the position of the robot 1. Thecommand generation unit 34 generates a movement command for driving therobot 1 based on the movement direction and the movement amount of theposition of the robot 1. The command generation unit 34 transmits amovement command to the operation control unit 43. The operation controlunit 43 causes the robot 1 to change its position based on the movementcommand.

Subsequently, the control proceeds to step 115. In the first control,steps 115 to 122 are repeated until the difference between the relativeposition amounts falls within the determination range. In the firstcontrol, even when the workpieces are not aligned by one controloperation, the workpieces can be gradually brought close to desiredpositions by repeating steps 115 to 122.

In the control of the present embodiment, it is not necessary tocalibrate the visual sensor coordinate system with respect to thereference coordinate system. Alternatively, it is not necessary topreviously obtain a Jacobian matrix in order to align the workpieces.Thus, the workpieces can be aligned by a simple method.

In the first control of the present embodiment, the position of thesecond workpiece 91 is adjusted in the period during which the firstworkpiece 81 is conveyed by the conveyor 75. A control operation fordetecting the first characteristic amount and the second characteristicamount by the characteristic amount detection unit 32, a controloperation for detecting the relative position amount by the calculationunit 33, and a control operation for calculating the movement command bythe command generation unit 34, are repeated. This control enables theposition of the robot 1 which grasps the workpiece 91 to follow theposition of the workpiece 81 which is conveyed by the conveyor 75.

Note that the camera 25 in the present embodiment is supported by thehand 2 via the support member 17. The hand 2 grasps the second workpiece91. Thus, the relative position and orientation of the workpiece 81 withrespect to the camera 25 are constant while the second workpiece 91 isaligned with the first workpiece 81. Referring to FIGS. 5, 7, and 8 ,even when the relative position of the second workpiece 91 with respectto the first workpiece 81 changes, the positions of the workpiece 91 andthe projection part 92 in the images 62 and 63 captured by the camera 25is constant. As a result, the position of the set point P2 of the secondcharacteristic portion in the image captured by the camera 25 isconstant.

Referring to FIG. 6 , in the control described above, every time thecamera 25 captures an image of the first workpiece 81 and an image ofthe second workpiece 91, the second characteristic portion of the secondworkpiece 91 is detected, and the second characteristic amount isdetected. Meanwhile, the coordinate value of the set point P2 which isthe second characteristic amount is constant, and accordingly, thestorage part 42 can store the second characteristic amount acquired fromthe image captured by the camera 25 for the first time. For the imagescaptured by the camera 25 for the second and subsequent times at steps115 to 117, the characteristic amount detection unit 32 can acquire thesecond characteristic amount from the storage part 42. Thecharacteristic amount detection unit 32 is only required to detect thefirst characteristic portion and the first characteristic amount. Thiscontrol is performed so that a control operation for calculating therelative position amount from the images captured by the camera for thesecond and subsequent times can be simplified. The processing time forthe images captured by the camera for the second and subsequent times isshortened.

In the meantime, in the present embodiment, the movement direction andmovement speed of the first workpiece 81 to be moved by the conveyor 75are previously determined. The robot controller 4 can performfeedforward control for changing the position of the robot 1 inaccordance with the movement of the workpiece 81 by the conveyor 75. Inthe present embodiment, the workpiece 81 moves at a constant movementspeed.

The command generation unit 34 calculates the movement direction and themovement speed of the position of the robot 1, in which the position ofthe robot 1 follows the position of the first workpiece 81 moved by theconveyor 75. For example, the command generation unit 34 calculates themovement direction so that the tool tip of the robot 1 moves in themovement direction of the workpiece 81. The command generation unit 34can calculate the movement amount at which the tool tip of the robot 1moves in the same direction as the movement direction of the workpiece81, at the same movement speed as the workpiece 81. Then, the commandgeneration unit 34 can control the movement direction and the movementamount of the position of the robot 1 based on the conveyance of theconveyor 75, and can further control the movement direction and themovement amount calculated based on the difference between the relativeposition amounts described above.

By adopting this control, the position and orientation of the robot 1related to the movement of the first workpiece 81 by the conveyor 75 canbe changed by feedforward control. In the control based on thedifference between the relative position amounts, it is only required tocorrect the relative positional deviation of the second workpiece 91with respect to the first workpiece 81, and accordingly, the secondworkpiece 91 can be aligned with the first workpiece 81 in a short time.

FIG. 9 is an enlarged perspective view of a hand, a first workpiece, anda second workpiece in a second robot device in the present embodiment.In the first robot device 5, the position of the robot 1 is adjusted byone camera 25. However, the embodiment is not limited to this. The robotdevice can adjust the position of the robot 1 using two or more cameras.

In the second robot device 8, a support member 18 is secured to the hand2. The support member 18 has an upward extending portion 18 a and adownward extending portion 18 b. As in the first robot device 5, acamera 25 as a first visual sensor is secured to the upward extendingportion 18 a. The camera 25 captures an image in an imaging range 25 a.A camera 26 as a second visual sensor is secured to the downwardextending portion 18 b. The camera 26 captures images of the firstworkpiece 81 and the second workpiece 91. In particular, the camera 26captures an image when the second workpiece 91 moves closer to the firstworkpiece 81. The camera 26 is arranged so as to be able to captureimages of the projection part 83 of the first workpiece 81 and theprojection part 93 of the second workpiece 91. The downward extendingportion 18 b supports the camera 26 so that the images of the projectionparts 83 and 93 can be captured from the sides of the workpieces 81 and91. The camera 26 captures an image in the imaging range 26 a. Thecamera 26 of the present embodiment is a two-dimensional camera.

The two cameras 25 and 26 are arranged so that their optical axes extendin different directions. In the present embodiment, the camera 26 isarranged so that the optical axis of the camera 26 extends in adirection substantially orthogonal to the optical axis of the camera 25.The second robot device 8 performs second control based on the imagescaptured by the two cameras 25 and 26. In the second control, theposition of the workpiece 91 with respect to the workpiece 81 isadjusted using the images captured by the cameras 25 and 26.

In the second control, a movement command is generated, based on theimage of the camera 25, by the first control. Further, the movementcommand is generated, based on the image of the camera 26, by a methodsimilar to that of the first control. In the present embodiment, a thirdcharacteristic portion for detecting the position of the first workpiece81 and a fourth characteristic portion for detecting the position of thesecond workpiece 91 are previously determined in the image captured bythe camera 26.

The third characteristic portion is different from the firstcharacteristic portion. As the third characteristic portion, forexample, a side surface of the projection part 83 of the first workpiece81 can be set. A third set point P3 for defining the position of thefirst workpiece 81 can be set in the third characteristic portion. Thefourth characteristic portion is different from the secondcharacteristic portion. As the fourth characteristic portion, forexample, a side surface of the projection part 93 of the secondworkpiece 91 can be set. A fourth set point P4 for defining the positionof the second workpiece 91 can be set in the fourth characteristicportion.

The characteristic amount detection unit 32 detects the thirdcharacteristic amount related to the position of the thirdcharacteristic portion and the fourth characteristic amount related tothe position of the fourth characteristic portion in the image capturedby the camera 26. In the image captured by the camera 26, the coordinatevalue of the set point P3 on the screen coordinate system 52 is thethird characteristic amount. Further, the coordinate value of the setpoint P4 on the screen coordinate system 52 is the fourth characteristicamount. The calculation unit 33 calculates, as the relative positionamount, the difference between the third characteristic amount and thefourth characteristic amount.

Further, the reference image related to the image captured by the camera26 when the second workpiece 91 is arranged at a target position withrespect to the first workpiece 81 is previously created. Further, therelative position amount in the reference image, which is the differencebetween the third characteristic amount and the fourth characteristicamount, is determined. The relative position amount in the referenceimage can be previously calculated.

The command generation unit 34 calculates the difference between therelative position amounts based on the relative position amount in theimage captured by the camera 26 and the relative position amount in thereference image including the images of the third characteristic portionand the fourth characteristic portion. The command generation unit 34then generates a movement command for operating the robot 1 so that thesecond workpiece 91 is arranged at a target position with respect to thefirst workpiece 81, based on the difference between the relativeposition amounts. The command generation unit 34 generates a movementcommand for operating the robot 1 so that the relative position amountin the image captured by the second camera 26 approaches the relativeposition amount in the reference image.

The command generation unit 34 can generate a final movement command tobe transmitted to the operation control unit 43, based on the movementcommand generated from the image of the camera 25 and the movementcommand generated from the image of the camera 26. For example, thecommand generation unit 34 can drive the robot 1 by one of the movementcommand based on the image of the camera 25 and the movement commandbased on the image of the camera 26, and then drive the robot 1 by theother movement command.

Alternatively, the movement command based on the image of the camera 25and the movement command based on the image of the camera 26 may becombined. For example, when the movement direction of the position ofthe robot 1 related to the u-axis in the image captured by the camera 25and the movement direction of the position of the robot 1 related to theu-axis in the image captured by the camera 26 match with each other, theaverage value of the movement amounts of the robot 1 may be calculated.The control for adjusting the position of the robot 1 can be repeateduntil the difference between the relative position amounts based on theimage captured by the camera 25 falls within the determination range,and the difference between the relative position amounts based on theimage captured by the camera 26 falls within the determination range.

In the second robot device 8, the cameras 25 and 26 capture images ofdifferent characteristic portions so as to perform position control.Thus, the position of the workpiece 91 can be adjusted more accuratelythan the control for adjusting the position using one camera. Further,the position of the workpiece can be adjusted based on the imagescaptured by a plurality of cameras without performing, for example,stereo measurement. Furthermore, the position of the workpiece can beadjusted by a plurality of two-dimensional cameras without using athree-dimensional camera.

Note that the position of the set point P2 in the image captured by thefirst camera 25 and the position of the set point P4 in the imagecaptured by the second camera 26 do not change. Thus, as in the firstcontrol, the second characteristic amount and the fourth characteristicamount which are detected at the beginning can be stored in the storagepart 42. In the control performed for the second and subsequent times,the second characteristic amount and the fourth characteristic amountwhich are stored in the storage part 42 may be acquired and the relativeposition amount in each image may be calculated.

In the meantime, in the second robot device 8, a plurality of cameras 25and 26 are arranged. In a controller for the robot device 8, theposition of the workpiece 91 can be adjusted and the orientation of theworkpiece 91 can be corrected. Subsequently, the correction of theorientation of the workpiece 91 will be described.

FIG. 10 is another enlarged perspective view of the hand, the firstworkpiece, and the second workpiece in the second robot device in thepresent embodiment. In the first control and the second control, therelative orientation of the second workpiece 91 with respect to thefirst workpiece 81 is adjusted before the position of the secondworkpiece 91 is adjusted. Meanwhile, the orientation of the secondworkpiece 91 may deviate from that of the first workpiece 81.

When the workpiece 91 takes a target orientation, the direction in whichthe pin 92 a and the pin 93 a are arranged is parallel to the directionin which the holes 82 a and 83 a are arranged. In the example shown inFIG. 10 , the orientation of the workpiece 91 is slightly rotated fromthe target orientation as indicated by arrow 106. The workpiece 91slightly rotates around the X-axis of the reference coordinate system 51in the direction from the Y-axis to the Z-axis. The workpiece 91slightly rotates around the rotation axis of the flange 16 from thetarget orientation. As a result, the projection part 92 of the workpiece91 is displaced from the projection part 82 of the workpiece 81 towardthe negative side of the Y-axis, as designated by arrow 107. Further,the projection part 93 of the workpiece 91 is displaced from theprojection part 83 of the workpiece 81 toward the positive side of theY-axis, as designated by arrow 108.

The plurality of the cameras 25 and 26 in the present embodiment areformed to capture images of the portions which are spaced from eachother in the second workpiece 91. The first camera 25 is arranged so asto capture an image of the projection part 92 as a portion arranged onone side of the second workpiece 91 in a predetermined direction.Further, the second camera 26 is arranged so as to capture an image ofthe projection part 93 as a portion arranged on the other side of thesecond workpiece 91 in the predetermined direction. The predetermineddirection corresponds to the direction in which the pins 92 a and 93 aare aligned.

FIG. 11 shows an image captured by the first camera when the orientationof the second workpiece deviates from that of the first workpiece.Referring to FIGS. 10 and 11 and FIG. 5 which is the reference image,the relative position of the workpiece 91 with respect to the workpiece81 in the image 66 is different from the relative position of theworkpiece 91 with respect to the workpiece 81 in the reference image 61.The projection part 92 of the workpiece 91 is displaced from theprojection part 82 of the workpiece 81 as indicated by arrow 107.Further, the extending direction of the pin 92 a secured to theprojection part 92 and the extending direction of the hole 82 a formedin the projection part 82 are not parallel to each other but aredifferent from each other.

FIG. 12 shows an image captured by the second camera when theorientation of the second workpiece with respect to the first workpieceis a target orientation. In the image 67, the extending direction of thepin 93 a secured to the projection part 93 and the extending directionof the hole 83 a formed in the projection part 83 are parallel. Thus, byperforming alignment of the workpiece 91 with the workpiece 81, the pin93 a can be inserted into the hole 83 a.

FIG. 13 shows an image captured by the second camera when theorientation of the second workpiece deviates from that of the firstworkpiece. In the image 68, the workpiece 91 slightly rotates withrespect to the workpiece 81 as indicated by arrow 108. Referring toFIGS. 10, 12, and 13 , the relative position of the projection part 93with respect to the projection part 83 in the image 68 is displaced asindicated by arrow 108. The extending direction of the pin 93 a is notparallel to the extending direction of the hole 83 a. Thus, even whenthe tip of the pin 93 a is arranged directly above the tip of the hole83 a, the pin 93 a cannot be inserted into the hole 83 a. In thirdcontrol of the present embodiment, the relative orientation of thesecond workpiece 91 with respect to the first workpiece 81 is corrected.

FIG. 14 is a flowchart of the third control in the second robot deviceof the present embodiment. In the third control, the orientation of theworkpiece 91 is corrected in addition to the second control.

Steps 131 to 138 are identical to the steps of the second control. Atstep 131, the first camera 25 and the second camera 26 capture images.At step 132, the characteristic amount detection unit 32 detects thefirst characteristic portion and the second characteristic portion inthe image captured by the first camera 25. At step 133, thecharacteristic amount detection unit 32 detects the third characteristicportion and the fourth characteristic portion in the image captured bythe second camera 26.

At step 134, the characteristic amount detection unit 32 detects thefirst characteristic amount related to the position of the firstcharacteristic portion, the second characteristic amount related to theposition of the second characteristic portion, and the thirdcharacteristic amount related to the position of the thirdcharacteristic portion, and the fourth characteristic amount related tothe position of the fourth characteristic portion. Referring to FIG. 11, the characteristic amount detection unit 32 detects, as the firstcharacteristic amount, the coordinate value of the set point P1 on thescreen coordinate system 52 in the image 66 captured by the first camera25. Further, the characteristic amount detection unit 32 detects, as thesecond characteristic amount, the coordinate value of the set point P2on the screen coordinate system 52. Referring to FIG. 13 , thecharacteristic amount detection unit 32 detects, as the thirdcharacteristic amount, the coordinate value of the set point P3 on thescreen coordinate system 52 in the image 68. The characteristic amountdetection unit 32 detects, as the fourth characteristic amount, thecoordinate value of the set point P4 on the screen coordinate system 52.

Referring to FIG. 14 , at step 135, the calculation unit 33 of the imageprocessing unit 31 calculates the relative position amount based on thefirst characteristic amount and the second characteristic amount. Atstep 136, the calculation unit 33 calculates the relative positionamount based on the third characteristic amount and the fourthcharacteristic amount. The calculation unit 33 calculates the relativeposition amount in the images 66 and 68 captured by the cameras 25 and26, respectively.

Subsequently, at step 137, regarding the images 66 and 68 respectivelycaptured by the cameras 25 and 26, the command generation unit 34 of theimage processing unit 31 calculates the difference between the relativeposition amounts, which is the difference between the relative positionamount in each of the images 66 and 68 and the relative position amountin the reference image. At step 138, the command generation unit 34generates a movement command for the robot 1, for each of the images 66and 68, based on the difference between the relative position amounts.The difference between the relative position amounts is calculated as avalue related to the u-axis and the v-axis in the screen coordinatesystem 52. The command generation unit 34 generates a movement commandfor the robot 1 based on the difference between the relative positionamounts.

The command generation unit 34 calculates the movement direction and themovement amount of the position of the robot 1 in the referencecoordinate system 51, based on the image 66 captured by the camera 25.Further, the command generation unit 34 calculates the movementdirection and the movement amount of the position of the robot 1 in thereference coordinate system 51 based on the image 68 captured by thecamera 26. In other words, the command generation unit 34 calculates themovement amount along the direction of the coordinate axis of thereference coordinate system 51 for each coordinate axis of the referencecoordinate system 51.

Subsequently, at step 139, the command generation unit 34 determineswhether the orientation of the second workpiece 91 with respect to thefirst workpiece 81 remains within a predetermined determination range,based on the movement command generated from the image captured by thefirst camera 25 and the movement command generated from the imagecaptured by the second camera 26.

In the present embodiment, the command generation unit 34 acquires themovement direction of the position of the robot 1 on a predeterminedcoordinate axis of the reference coordinate system 51. Referring to FIG.10 , in this example, the Y-axis is previously selected from thecoordinate axes of the reference coordinate system 51. The commandgeneration unit 34 acquires a Y-axis movement command of the movementcommands generated from the image 66 captured by the first camera 25 anda Y-axis movement command of the movement commands generated from theimage 68 captured by the second camera 26.

When Y-axis movement directions acquired from the images 66 and 68captured by the two cameras 25 and 26 are the same, the commandgeneration unit 34 determines that the orientation of the secondworkpiece 91 with respect to the first workpiece 81 remains within apredetermined determination range. Meanwhile, the command generationunit 34 determines that the orientation of the second workpiece 91 withrespect to the first workpiece 81 deviates from the determination rangewhen the Y-axis movement directions are different from each other.

In the examples shown in FIGS. 10 to 13 , in the movement command basedon the image captured by the first camera 25, a command for moving therobot 1 in a direction opposite to the direction designated by arrow 107is generated. Further, in the movement command based on the imagecaptured by the second camera 26, a command for moving the robot 1 in adirection opposite to the direction designated by arrow 108 isgenerated. Thus, at step 139, the command generation unit 34 determinesthat the Y-axis movement directions based on the images captured by thetwo cameras are different from each other. In this respect, the controlproceeds to step 140.

Note that, in the present embodiment, in a predetermined coordinate axisof the reference coordinate system, the orientation of the workpiece isdetermined to be within the determination range when the directions inwhich the position of the robot should move are the same. However, theembodiment is not limited to this. Even when the movement directions ona predetermined coordinate axis are different from each other, theorientation of the workpiece may be determined to be within thedetermination range if the movement amount is minute.

At step 140, the command generation unit 34 sets a method for correctingthe orientation of the robot 1. The command generation unit 34 generatesa movement command for rotating the workpiece 91 in a direction oppositeto the direction in which the second workpiece 91 is inclined withrespect to the first workpiece 81. Referring to FIG. 10 , in the presentembodiment, the workpiece 91 is controlled so as to rotate, in thedirection from the Z-axis to the Y-axis, around the X-axis of thereference coordinate system 51. An orientation correction amount θ whichis the rotation amount of the workpiece 91 can be previously determined.

At step 141, the command generation unit 34 transmits a movement commandbased on the method for correcting the orientation of the robot 1 to theoperation control unit 43. The operation control unit 43 corrects theorientation of the robot based on the movement command received from thecommand generation unit 34.

Subsequently, at step 142, the image processing unit 31 corrects themovement direction of the position of the robot 1 in the referencecoordinate system 51 with respect to the coordinate axis of the screencoordinate system 52. The difference between the relative positionamounts is calculated by the value related to the u-axis and the v-axisof the screen coordinate system 52. The movement direction of theposition of the robot 1 in the reference coordinate system 51, whichcorresponds to the u-axis and the v-axis of the screen coordinate system52, is previously determined. When the orientation of the robot 1 iscorrected, the movement direction in the reference coordinate system 51,which corresponds to the u-axis and the v-axis of the images captured bythe cameras 25 and 26, also changes. In other words, the movementdirection of the position of the robot 1 represented by the coordinatevalues of the X-axis, Y-axis, and Z-axis of the reference coordinatesystem changes.

The image processing unit 31 corrects the movement direction of theposition of the robot 1, which corresponds to the coordinate axis of thescreen coordinate system 52, based on the correction amount of theorientation of the robot 1. For example, a transformation matrixcalculated based on the correction amount of the orientation of therobot 1 may be multiplied by the coordinate values of the X-axis,Y-axis, and Z-axis of the reference coordinate system 51, which indicatethe movement direction of the position of the robot 1, and whereby themovement direction of the position of the robot 1 can be corrected.

After step 142, the control returns to step 131. The control at steps131 to 139 is then repeated. At step 139, when the movement directionsin a predetermined coordinate axis of the reference coordinate system 51are different from each other, the control at steps 140, 141, and 142 isperformed again.

At step 140, a method for correcting the orientation of the robot 1 isset. When the orientation of the robot 1 is corrected by the previousorientation correction amount θ, the direction in which the orientationof the second workpiece 91 deviates from that of the first workpiece 81may be reversed. In this respect, the orientation of the robot 1 iscorrected in the direction opposite to the direction in the previousorientation correction. Furthermore, a correction amount which issmaller than the previous correction amount is adopted. For example, theorientation is controlled so as to be changed by a correction amount(-θ/2) which is half the previous correction amount. Meanwhile, evenwhen the orientation of the robot 1 is corrected by the previousorientation correction amount θ, the direction in which the orientationof the second workpiece 91 deviates from that of the first workpiece 81may be the same as the direction in the previous control. In thisrespect, the orientation is corrected in the same direction and the sameorientation correction amount θ as the previous one. After that, thecontrol at steps 131 to 139 can be repeated again.

In this way, when the correction of the orientation in the previouscontrol causes the direction in which the orientation deviates to bereversed, the orientation of the robot can be corrected so as to bereversed to the opposite direction and the correction amount can bereduced. This control enables the deviation of the orientation of thesecond workpiece 91 with respect to the first workpiece 81 to begradually corrected. In the images captured by the first camera 25 andthe second camera 26, the orientation of the robot 1 can be correcteduntil the directions of the second workpiece 91 and the first workpiece81 which are misaligned with each other are equalized.

Note that, at step 139, when the orientation of the second workpiecewith respect to the first workpiece deviates from the determinationrange, any control can be performed. For example, the robot device maybe stopped. Further, at step 140, the command generation unit 34 can seta method for correcting the orientation of the robot 1 by any control.For example, the orientation correction amount of the workpiece 81 maybe calculated based on the movement amount in the Y-axis directioncalculated from the images captured by the two cameras 25 and 26.

At step 139, when the movement direction based on the image captured bythe first camera 25 and the movement direction based on the imagecaptured by the second camera 26 are the same, the control proceeds tostep 143.

At steps 143 to 145, as in the second control, the position of the robot1 is determined and corrected. At step 143, it is determined whethereach of the difference between the relative position amounts based onthe image captured by the first camera 25 and the difference between therelative position amounts based on the image captured by the secondcamera 26 remains within the determination range. At step 143, when atleast one of the differences between the relative position amountsdeviates from the determination range, the control proceeds to step 144.

At step 144, the command generation unit 34 generates a final movementcommand for the robot based on the movement command based on each image.At step 145, the operation control unit 43 then drives the robot 1 basedon the final movement command for the robot so as to change the positionof the robot 1. After that, the control proceeds to step 131.

At step 143, when the differences between the relative position amountsin the two images remain within the determination range, the positionand orientation of the second workpiece 91 with respect to the firstworkpiece 81 can be determined to be the target position andorientation. The control can then end.

The configuration, action, and effect of the second robot device 8 otherthan those described above are the same as those of the first robotdevice 5, and thus, the description thereof will not be repeated below.

FIG. 15 is an enlarged perspective view of a hand, a first workpiece,and a second workpiece of a third robot device in the presentembodiment. In the third robot device 9, a support member 19 is securedto the hand 2. The support member 19 has an upward extending portion 19a and a downward extending portion 19 b. As in the second robot device8, a camera 25 serving as a first visual sensor is secured to the upwardextending portion 19 a. A camera 26 serving as a second visual sensor issecured to the downward extending portion 19 b.

The direction of the camera 26 of the third robot device 9 is differentfrom the direction of the camera 26 of the second robot device 8. Thedownward extending portion 19 b of the support member 19 supports thecamera 26 so that the imaging range 26 a of the camera 26 overlaps theimaging range 25 a of the camera 25. The camera 26 of the third robotdevice 9 is arranged so as to capture an image of the same portion asthe portion captured by the camera 25. The camera 26 is arranged so asto be able to capture images of the projection part 82 of the workpiece81 and the projection part 92 of the workpiece 91.

In the third robot device 9, fourth control is performed in order toalign the workpiece 91 with the workpiece 81. In the fourth control, asin the second control for the second robot device 8, a thirdcharacteristic portion and a fourth characteristic portion are set, andthe position of the robot 1 is adjusted based on the images captured bythe cameras 25 and 26. In this embodiment, a side surface of theprojection part 82 is set as the third characteristic portion. A sidesurface of the projection part 92 is set as the fourth characteristicportion. In the third robot device 9, the third characteristic portionmay be the same as or different from the first characteristic portion.Further, the fourth characteristic portion may be the same as ordifferent from the second characteristic portion.

The configuration, action, and effect of the third robot device 9 otherthan those described above are similar to those of the second robotdevice 8, and thus, the description thereof will not be repeated below.

The controller for the second robot device 8 and the controller for thethird robot device 9 include two cameras, but the embodiment is notlimited to this. The controller for the robot device may include threeor more cameras. The controller may cause each camera to capture animage of the corresponding characteristic portion different from theother characteristic portions, and may control the position andorientation of the workpiece based on the images captured by thecameras.

FIG. 16 is an enlarged perspective view of a hand, a first workpiece,and a second workpiece in a fourth robot device in the presentembodiment. The fourth robot device 10 includes a third camera 28 inaddition to the first camera 25 and the second camera 26. The fourthrobot device 10 has a support member 20 supported by the hand 2. Thesupport member 20 has an upward extending portion 20 a which supportsthe camera 25 and a downward extending portion 20 b which supports thecamera 26. Further, the support member 20 has a laterally extendingportion 20 c which supports the camera 28. The camera 28 has an imagingrange 28 a. The camera 28 is arranged so that its optical axis extendsin a direction different from the direction of the optical axis of thecamera 25 and the direction of the optical axis of the camera 26.

In the fourth robot device 10, fifth control for adjusting the positionand orientation of the workpiece 91 with respect to the workpiece 81 isperformed. In the fifth control, a fifth characteristic portion is seton the first workpiece 81 in order to process an image captured by thecamera 28. Further, a sixth characteristic portion is set on the secondworkpiece 91. For example, a side surface of the projection part 83 ofthe workpiece 81 is set as the fifth characteristic portion, and a sidesurface of the projection part 93 of the workpiece 91 is set as thesixth characteristic portion.

The same control as the first control can be performed for an imagecaptured by the camera 28. The camera 28 is arranged so as to be able tocapture images of the fifth characteristic portion and the sixthcharacteristic portion. A fifth set point and a sixth set point fordefining the positions of the characteristic portions are set in thefifth characteristic portion and the sixth characteristic portion. Thecharacteristic amount detection unit 32 can detect a fifthcharacteristic amount corresponding to the fifth characteristic portionand a sixth characteristic amount corresponding to the sixthcharacteristic portion, based on the position of the set points in thescreen coordinate system 52. The reference image for the camera 28 ispreviously created. The fifth characteristic amount and the sixthcharacteristic amount in the reference image can be previouslycalculated. Further, the relative position amount in the reference imagecan be previously calculated.

In the fifth control, the same control as the third control can beperformed. In the fifth control, the position and orientation of thesecond workpiece 91 with respect to the first workpiece 81 are adjustedby three cameras. In the control for adjusting the position of theworkpiece 91 with respect to the workpiece 81, the deviation of theposition of the second workpiece 91 from that of the first workpiece 81can be detected based on the images captured by the three cameras 25,26, and 28. Thus, the position can be adjusted more accurately than theposition which is adjusted by two cameras.

Further, in the control for correcting the orientation of the workpiece91 with respect to the workpiece 81, the orientation of the workpiece 91which is displaced around the Y-axis of the reference coordinate system51 can be corrected based on the image captured by the camera 25 and theimage captured by the camera 28. In this way, the number of cameras isincreased, and whereby the number of direction in which the deviation oforientation is corrected can be increased.

The configuration, action, and effect of the fourth robot device otherthan those described above are similar to those of the first robotdevice 5, the second robot device 8, and the third robot device 9, andthus, the description thereof will not be repeated below.

In the first robot device 5, the second robot device 8, the third robotdevice 9, and the fourth robot device 10, the cameras 25, 26, and 28 aresecured to the hand 2 and move together with the hand 2, but theembodiment is not limited to this. The cameras may be secured to theinstallation surface, etc. In other words, the cameras may be secured sothat the position and orientation of the cameras do not change even whenthe position and orientation of the robot 1 change.

FIG. 17 is a schematic view of a fifth robot device in the presentembodiment. The fifth robot device 6 arranges the workpiece 97 servingas the second member inside the case 87 serving as the first member. Thecase 87 corresponds to the first workpiece, and the workpiece 97corresponds to the second workpiece. The fifth robot device 6 has a hand7 attached to the robot 1. The hand 7 is formed to grasp the workpiece97 by suction. The fifth robot device 6 includes a conveyor 75 as acarrier which conveys the case 87.

The fifth robot device 6 has a camera 27 as a visual sensor, which issecured to the installation surface via a mount 71. The camera 27remains stationary even when the position and orientation of the robot 1change. The camera 27 is spaced from the conveyor 75 by a sufficientdistance so that the images of the workpiece 97 and the case 87 can becaptured while the robot 1 is adjusting the position of the workpiece97.

FIG. 18 is a side view of the fifth robot device when a workpiece istried to be accommodated in a case. The case 87 is conveyed by theconveyor 75 as designated by arrow 105. The case 87 has a plurality ofwalls 87 a formed corresponding to the shape of the workpiece 97. Thefifth robot device 6 accommodates the workpiece 97 in a regionsurrounded by the walls 87 a. In the example shown in FIG. 18 , theworkpiece 97 is aligned with the region surrounded by the walls 87 a.Thus, the workpiece 97 moves downward as designated by arrow 104 and canbe accommodated in the case 87.

FIG. 19 is another side view of the fifth robot device when a workpieceis tried to be accommodated in a case. In the example shown in FIG. 19 ,the workpiece 97 is misaligned with the region surrounded by the walls87 a. The fifth robot device 6 performs sixth control. In the sixthcontrol, the position of the robot 1 is adjusted so that the workpiece97 is placed directly above the region surrounded by the walls 87 a. Theorientation of the workpiece 97 with respect to the case 87 ispreviously adjusted. In the sixth control, the displacement of theposition of the workpiece 97 from the position of the case 87 iscorrected by the same control as the first control. As shown in FIG. 18, the reference image, in which the workpiece 97 is aligned with thecase 87, is previously created.

FIG. 20 shows an image captured by the camera. In the image of FIG. 20 ,the workpiece 97 is misaligned with the region surrounded by the walls87 a. In the sixth control, for example, a side surface of each wall 87a can be set as the first characteristic portion. A first set point P1can be set at a corner of the side surface of the wall 87 a. Further, aside surface of the workpiece 97 can be set as the second characteristicportion. A second set point P2 can be set at a corner of the sidesurface of the workpiece 97.

Referring to FIGS. 3 and 20 , the characteristic amount detection unit32 detects the first characteristic amount related to the position ofthe first characteristic portion and the second characteristic amountrelated to the position of the second characteristic portion. Thecalculation unit 33 calculates the relative position amount based on thefirst characteristic amount and the second characteristic amount. Thecommand generation unit 34 can generate a movement command for operatingthe robot 1, based on the relative position amount in the image 65captured by the camera 27 and the relative position amount in thereference image.

Even in a robot device with a fixed camera, the position of the robotcan be controlled by the same control as the first control. Further, thesame control as the second control, the third control, or the fourthcontrol can be performed using a plurality of cameras secured to, forexample, the installation surface.

The configuration, action, and effect of the fifth robot device 6 otherthan those described above are similar to those of the first robotdevice 5, the second robot device 8, the third robot device 9, and thefourth robot device 10, and thus, the description thereof will not berepeated below.

The visual sensor in the above embodiment is a two-dimensional camera,but the embodiment is not limited to this. The visual sensor may be athree-dimensional camera capable of detecting the three-dimensionalposition of a member included in the image. The use of thethree-dimensional camera as the visual sensor enables the position ofthe first member and the position of the second member in the referencecoordinate system to be detected without the base image.

In the above embodiments, the robot device for assembling a product andthe robot device for inserting a workpiece into a case are adopted.However, the embodiment is not limited to this. The controller of thepresent embodiment can be applied to a robot device which performs anyoperation. The controller of the present embodiment can be applied to,for example, a robot device which removes a workpiece from a case, arobot device which performs spot welding, or a robot device whichapplies an adhesive agent.

The first member and the second member in the above embodiments areworkpieces, or a workpiece and a case, but the embodiment is not limitedto this. The first member includes any member for which the robot deviceperforms an operation. Further, the second member includes any memberwhich is moved by driving the robot. In particular, the second membermay be an operation tool attached to the robot. For example, in a robotdevice which performs spot welding, a spot-welding gun as an operationtool is attached to the robot. In this respect, the second member is thespot-welding gun. In the spot-welding gun, the second characteristicportion, the fourth characteristic portion, and the sixth characteristicportion can be set as the portions which can be detected in the imagecaptured by the camera.

In the above embodiments, the characteristic portions are set on thesurface of a workpiece or a case. However, the embodiment is not limitedto this. The characteristic portions may be set at components of therobot, components of the hand, or components of the carrier. Forexample, in the first to fourth control, no displacement of theworkpiece in the hand may occur. Alternatively, the position of therobot may be adjusted in accordance with the displacement of theworkpiece in the hand. In these respects, the second characteristicportion, the fourth characteristic portion, and the sixth characteristicportion may be set in the hand when the hand is arranged within theimaging range of the camera.

According to the present disclosure, it is possible to provide acontroller for a robot device which accurately controls the position ofa robot by a simple method.

In each control described above, the order of steps can be appropriatelychanged as far as the function and action are not changed.

The above embodiments can be combined appropriately. In each of thedrawings described above, the same or similar parts are designated bythe same reference numerals. Note that the above embodiments aredescribed for illustrative purposes only, and do not limit theinvention. Further, the embodiments include modifications of theembodiments described in the claims.

1. A controller for a robot device in which a robot moves a secondmember so as to adjust a relative position of the second member withrespect to a first member, comprising: a visual sensor mounted on therobot; and a processor configured to: in a first control, obtain areference image captured by the visual sensor in a state where thesecond member is arranged at a target position with respect to the firstmember, the reference image including images of a first characteristicportion of the first member and a second characteristic portion of thesecond member; detect a position of the first characteristic portion ofthe first member and a position of the second characteristic portion ofthe second member in the reference image; and calculate, as a firstrelative position amount, a difference between coordinate values of theposition of the first characteristic portion and coordinate values ofthe position of the second characteristic portion in the referenceimage, the processor is further configured to: in a second control,obtain an image captured by the visual sensor in a state where thesecond member is arranged with respect to the first member, the imageincluding images of the first characteristic portion of the first memberand the second characteristic portion of the second member; detect aposition of the first characteristic portion of the first member and aposition of the second characteristic portion of the second member inthe image; calculate, as a second relative position amount, a differencebetween coordinate values of the first characteristic portion andcoordinate values of the second characteristic portion in the image;generate a movement command for operating the robot to move the secondmember to the target position with respect to the first member based onthe first relative position amount and the second relative positionamount; and operate the robot so that a position of the second member isadjusted to the target position with respect to the first member basedon the movement command, wherein in the image captured by the visualsensor mounted on the robot, the coordinate values of the position ofthe second characteristic portion are constant, the processor is furtherconfigured to: repeat the second control so that the position of thesecond member approaches the target position with respect to the firstmember; store the coordinate values of the position of the secondcharacteristic portion in a storage device when the position of thesecond characteristic portion is detected in the image captured by thevisual sensor for a first time in repetition of the second control; anduse the coordinate values of the position of the second characteristicportion stored in the storage device to calculate the second relativeposition amount for images captured by the visual sensor for second andsubsequent times in the repetition of the second control.
 2. Thecontroller according to claim 1, wherein the processor calculates, inthe second control, a difference between the first relative positionamount of the reference image and the second relative position amount inthe image, a movement direction of the position of the robot withrespect to the difference between the first relative position amount andthe second relative position amount is previously determined, a methodfor calculating a movement amount of the position of the robot withrespect to the difference between the first relative position amount andthe second relative position amount is previously determined, and theprocessor sets the movement direction and a movement amount of theposition of the robot based on the difference between the first relativeposition amount and the second relative position amount, and generatesthe movement command based on the movement direction and the movementamount of the position of the robot.
 3. The controller according toclaim 1, wherein the robot device includes a carrier for conveying thefirst member, and the processor repeats the second control while thecarrier conveys the first member.
 4. The controller according to claim3, wherein a movement direction and movement speed of the first memberconveyed by the carrier are previously determined, and the processorgenerates the movement command based on the movement direction and apredefined movement amount of the position of the robot, in which theposition of the robot follows the position of the first member conveyedby the carrier.
 5. The controller according to claim 1, wherein thevisual sensor constitutes a first visual sensor, the controller furthercomprises a second visual sensor mounted on the robot, wherein theprocessor is further configured to: in a third control, obtain a secondreference image captured by the second visual sensor in a state wherethe second member is arranged at the target position with respect to thefirst member, the second reference image including images of a thirdcharacteristic portion of the first member and a fourth characteristicportion of the second member; detect a position of the thirdcharacteristic portion of the first member and a position of the fourthcharacteristic portion of the second member in the second referenceimage; and calculate, as a third relative position amount, a differencebetween coordinate values of the position of the third characteristicportion and coordinate values of the position of the fourthcharacteristic portion in the second reference image, the processor isfurther configured to: in a fourth control, obtain an image captured bythe second visual sensor in a state where the second member is arrangedwith respect to the first member, the image including images of thethird characteristic portion of the first member and the fourthcharacteristic portion of the second member; detect a position of thethird characteristic portion of the first member and a position of thefourth characteristic portion of the second member in the image capturedby the second visual sensor; calculate, as a fourth relative positionamount, a difference between coordinate values of the thirdcharacteristic portion and coordinate values of the fourthcharacteristic portion in the image captured by the second visualsensor; generate a movement command for operating the robot to move thesecond member to the target position with respect to the first memberbased on the first relative position amount, the second relativeposition amount, the third relative position amount and the fourthrelative position amount; and operate the robot so that a position ofthe second member is adjusted to the target position with respect to thefirst member based on the movement command, wherein in the imagecaptured by the second visual sensor mounted on the robot, thecoordinate values of the position of the fourth characteristic portionare constant, the processor is further configured to: repeat the fourthcontrol so that the position of the second member approaches the targetposition with respect to the first member, store the coordinate valuesof the position of the fourth characteristic portion in the storagedevice when the position of the fourth characteristic portion isdetected in the image captured by the second visual sensor for a firsttime in repetition of the fourth control, use the coordinate values ofthe position of the fourth characteristic portion stored in the storagedevice to calculate the fourth relative position amount for imagescaptured by the second visual sensor for second and subsequent times inthe repetition of the fourth control.
 6. The controller according toclaim 5, wherein the first visual sensor is arranged so as to capture animage of a portion arranged on one side of the second member in apredetermined direction, the second visual sensor is arranged so as tocapture an image of a portion arranged on another side of the secondmember in the predetermined direction, wherein the predetermineddirection corresponds to a direction in which the portions of the secondmember are aligned, and the processor determines whether an orientationof the second member with respect to the first member remains within apredetermined determination range based on the movement commandgenerated from images captured by the first visual sensor and themovement command generated from images captured by the second visualsensor.